Relevance and Research Question: Grid or matrix questions pose problems for mobile devices (smartphones). Several survey software systems optimize for mobile devices by turning grids into item-by-item scrolling pages, but there is little empirical research to support this decision. We exploit a quasi-experimental design in which the same questions were asked in two different formats in two successive years to see if the item-by-item alternative takes less time to complete on smartphones than grids, and if data quality (specifically, missing data) is affected.

Methods and Data: We use data from a student Web survey. The 2013 survey used a traditional grid layout for many pages. The 2014 survey used a new version of mobile optimization from the vendor that turned grids into item-by-item scrolling pages. A total of 12 grid questions were identical in the two surveys. About 11% of the 2013 survey respondents (n=3,223) used a smartphone, while 22% did so in 2014 (n=4,233). This paper builds on analyses of the 2012 and 2013 surveys presented at GOR last year.

Results: Grid pages took on average 9% longer for smartphone users (26.93 secs) than PC users (mean=24.62 secs) in 2013. However, the stacked item-by-item pages in 2014 took on average 21% longer for smartphone users (mean=29.97 secs) than PC users (mean=24.86 seconds). Comparing the same set of 12 grid questions in 2013 and 2014, the stacked version took about 6% longer than the grid version (mean=28.8 in 2013 versus 30.51 in 2014) for smartphone users. We find no differences in missing data rates between the two versions.

Added Value: There is great interest in finding alternatives to grid questions for mobile devices, and software vendors are offering solutions without research evidence. This research provides some insight into this important practical question. We provide the first evidence that the stacked item-by-item alternative may not improve efficiency for smartphone users over traditional grid questions.